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ABSTRACT 


i  Experimentation  In  software  engineering  supports  the  advancement  of  the  field 

I  '  ,  \ 

i  through  an  Iterative  learning  process.  Ha  this  paper^we  present, a  framework  for  analyz¬ 

ing  most  of  the  experimental  work  performed  In  software  engineering  over  the  past 
several  years.  We  describe  a  variety  of  experiments  In  the  framework  and  discuss  their 
contribution  to  the  software  engineering  discipline.  Some  useful  recommendations  for 
the  application  of  the  experimental  process  In  software  engineering  are  included.  _ 
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1.  Introduction 


As  any  area  matures,  there  Is  the  need  to  understand  Its  components  and  their  rela¬ 
tionships.  An  experimental  process  provides  a  basis  for  the  needed  advancement  In 
knowledge  and  understanding.  Since  software  engineering  Is  In  Its  adolescence,  It  Is  cer¬ 
tainly  a  candidate  for  the  experimental  method  of  analysis.  Experimentation  Is  per¬ 
formed  In  order  to  help  us  better  evaluate,  predict,  understand,  control,  and  Improve 
the  software  development  process  and  product. 

Experimentation  In  software  engineering,  as  with  any  other  experimental  procedure, 
involves  an  Iteration  of  a  hypothesize  and  test  process.  Models  of  the  software  process 
or  product  are  built,  hypotheses  about  these  models  are  tested,  and  the  Information 
learned  Is  used  to  refine  the  old  hypotheses  or  develop  new  ones.  In  an  area  like  soft¬ 
ware  engineering,  this  approach  takes  on  special  Importance  because  we  greatly  need  to 
Improve  our  knowledge  of  how  software  Is  developed,  the  effect  of  various  technologies, 
and  what  areas  most  need  Improvement.  There  Is  a  great  deal  to  be  learned  and  intui¬ 
tion  Is  not  always  the  best  teacher. 

In  this  paper  we  lay  out  a  framework  for  analyzing  most  of  the  experimental  work 
that  has  been  performed  In  software  engineering  over  the  past  several  years.  We  then 
discuss  a  variety  of  these  experiments,  their  results,  and  the  Impact  they  have  had  on 
our  knowledge  of  the  software  engineering  discipline. 

2.  Objectives 

There  are  three  overall  goals  for  this  work.  The  first  objective  Is  to  describe  a 
framework  for  experimentation  In  software  engineering.  The  framework  for  experlmen- 


tatlon  Is  Intended  to  help  structure  the  experimental  process  and  to  provide  a 
classification  scheme  for  understanding  and  evaluating  experimental  studies.  The 
second  objective  Is  to  classify  and  discuss  a  variety  of  experiments  from  the  literature 
according  to  the  framework.  The  description  of  several  software  engineering  studies  Is 
Intended  to  provide  an  overview  of  the  knowledge  resulting  from  experimental  work,  a 
summary  of  current  research  directions,  and  a  basis  for  learning  from  past  experience 
with  experimentation.  The  third  objective  Is  to  Identify  problem  areas  and  lessons 
learned  In  experimentation  In  software  engineering.  The  presentation  of  problem  areas 
and  lessons  learned  Is  Intended  to  focus  attention  on  general  trends  in  the  field  and  to 
provide  the  experimenter  with  useful  recommendations  for  performing  future  studies. 
The  following  three  sections  address  these  goals. 

3.  Experimentation  Framework 

The  framework  of  experimentation,  summarized  In  Figure  1,  consists  of  four 
categories  corresponding  to  phases  of  the  experimentation  process:  I)  definition,  II)  plan¬ 
ning,  in)  operation,  and  IV)  Interpretation.  The  following  sections  discuss  each  of  these 
four  phases. 

3.1.  Experiment  Definition 

The  first  phase  of  the  experimental  process  Is  the  study  definition  phase.  The 
study  definition  phase  contains  six  parts:  A)  motivation,  B)  object,  C)  purpose,  D)  per¬ 
spective,  E)  domain,  and  F)  scope.  Most  study  definitions  contain  each  of  the  six  parts: 


an  example  definition  appears  In  Figure  2. 


There  can  be  several  motivations,  objects,  purposes,  or  perspectives  In  an  experi¬ 
mental  study.  For  example,  the  motivation  of  a  study  may  be  to  understand,  assess,  or 
Improve  the  effect  of  a  certain  technology.  The  ‘‘object  of  study"  Is  the  primary  entity 
examined  In  a  study.  A  study  may  examine  the  final  software  product,  a  development 
process  (e.g..  Inspection  process,  change  process),  a  model  (e.g.,  software  reliability 
model),  etc.  The  purpose  of  a  study  may  be  to  characterize  the  change  In  a  system  over 
time,  to  evaluate  the  effectiveness  of  testing  processes,  to  predict  system  development 
cost  by  using  a  cost  model,  to  motivate1  the  validity  of  a  theory  by  analyzing  empirical 
evidence,  etc.  In  experimental  studies  that  examine  "software  quality,"  the  Interpreta¬ 
tion  usually  Includes  correctness  If  It  Is  from  the  perspective  of  a  developer  or  reliability 
If  It  Is  from  the  perspective  of  a  customer.  Studies  that  examine  metrics  for  a  given  pro¬ 
ject  type  from  the  perspective  of  the  project  manager  may  Interest  certain  project 
managers,  while  corporate  managers  may  only  be  Interested  If  the  metrics  apply  across 
several  project  types. 

Two  Important  domains  that  are  considered  In  experimental  studies  of  software  are 
1)  the  Individual  programmers  or  programming  teams  (the  "teams")  and  11)  the  programs 
or  projects  (the  "projects").  "Teams"  are  (possibly  single-person)  groups  that  work 
separately,  and  "projects"  are  separate  programs  or  problems  on  which  teams  work. 
Teams  may  be  characterized  by  experience,  size,  organization,  etc.,  and  projects  may  be 
characterized  by  size,  complexity,  application,  etc.  A  general  classification  of  the  scopes 

1  For  clarification,  the  usage  of  the  word  "motivate"  as  a  study  purpose  Is  distinct 
from  the  study  "motivation." 
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of  experimental  studies  can  be  obtained  by  examining  the  sizes  of  these  two  domains 
considered  (see  Figure  3).  Blocked  subject-project  studies  examine  one  or  more  objects 
across  a  set  of  teams  and  a  set  of  projects.  Replicated  project  studies  examine  object(s) 
across  a  set  of  teams  and  a  single  project,  while  multi-project  variation  studies  examine 
object(s)  across  a  single  team  and  a  set  of  projects.  Single  project  studies  examine 
object(s)  on  a  single  team  and  a  single  project.  As  the  representativeness  of  the  samples 
examined  and  the  scope  of  examination  Increase,  the  wider-reaching  a  study’s  conclu¬ 
sions  become. 

3.2.  Experiment  Planning 

The  second  phase  of  the  experimental  process  Is  the  study  planning  phase.  The  fol¬ 
lowing  sections  discuss  aspects  of  the  experiment  planning  phase:  A)  design,  B)  criteria, 
and  C)  measurement. 

The  design  of  an  experiment  couples  the  study  scope  with  analytical  methods  and 
Indicates  the  domain  samples  to  be  examined.  Fractional  factorial  or  randomized  block 
designs  usually  apply  In  blocked  subject-project  studies,  while  completely  randomized  or 
Incomplete  block  designs  usually  apply  In  multi-project  and  replicated  project  studies 
[33,  40].  Multivariate  analysis  methods,  Including  correlation,  factor  analysis,  and  re¬ 
gression  [75,  80,  89],  generally  may  be  used  across  all  experimental  scopes.  Statistical 
models  may  be  formulated  and  customized  as  appropriate  [89].  Non-parametrlc 
methods  should  be  planned  when  only  limited  data  may  be  available  or  distributional 
assumptions  may  not  be  met  [99].  Sampling  techniques  [41]  may  be  used  to  select 
representative  programmers  and  prog  rams/ projects  to  examine. 


Different  motivations,  objects,  purposes,  perspectives,  domains,  and  scopes  require 
the  examination  of  different  criteria.  Criteria  that  tend  to  be  direct  reflections  of 
cost/quallty  Include  cost  [ill,  106,  86,  4,  28],  errors/changes  [49,  14,  109,  2,  81,  19],  reli¬ 
ability,  [42,  84,  56,  70,  69,  76,  77,  95],  and  correctness  [51  61,  88].  Criteria  that  tend  to 
be  Indirect  reflections  of  cost/quallty  Include  data  coupling  [62,  48,  102,  78],  Information 
visibility  [85,  83,  55],  programmer  understanding  [98,  100,  107,  1 10],  execution  coverage 
[103,  21,  24],  and  slze/complexlty  [17,  59,  7lJ. 

The  concrete  manifestations  of  the  cost/quallty  aspects  examined  In  the  experiment 
are  captured  through  measurement.  Paradigms  assist  In  the  metric  definition  process: 
the  goal-questlon-metrlc  paradigm  [20,  22,  25,  93]  and  the  factor-crlterla-metrlc  para¬ 
digm  [39,  72j.  Once  appropriate  metrics  have  been  de lined,  they  may  be  validated  to 
show  that  they  capture  what  Is  Intended  [12,  18,  44,  50,  106,  113).  The  data  collection 
process  Includes  developing  automated  collection  schemes  [15]  and  designing  and  testing 
data  collection  forms  [22,  10].  The  required  data  may  Include  both  objective  and  sub¬ 
jective  data  and  dlfferents  levels  of  measurement:  nominal  (or  classlflcatory),  ordinal  (or 
ranking).  Interval,  or  ratio  [99]. 

3.3.  Experiment  Operation 

The  third  phase  of  the  experimental  process  Is  the  study  operation  phase.  The 
operation  of  the  experiment  consists  of  A)  preparation,  B)  execution,  and  C)  analysis. 
Before  conducting  the  actual  experiment,  preparation  may  Include  a  pilot  study  to 
confirm  the  experimental  scenario,  help  organize  experimental  factors  (e.g.,  subject  ex¬ 


pertise),  or  Inoculate  the  subjects  [44,  43.  63.  24.  110.  73].  Experimenters  collect  and 


validate  the  defined  data  during  the  execution  of  the  study  [18,  109).  The  analysis  of 
the  data  may  Include  a  combination  of  quantitative  and  qualitative  methods  (30).  The 
preliminary  screening  of  the  data,  probably  using  plots  and  histograms,  usually  proceeds 
the  formal  data  analysis.  The  process  of  analyzing  the  data  requires  the  Investigation  of 
any  underlying  assumptions  (e.g.,  distributional)  before  the  application  or  the  statistical 
models  and  tests. 

3.4.  Experiment  Interpretation 

The  fourth  phase  of  the  experimental  process  Is  the  study  Interpretation  phase. 
The  Interpretation  of  the  experiment  consists  of  A)  Interpretation  context,  B)  extrapola¬ 
tion,  and  C)  Impact.  The  results  of  the  data  analysis  rrom  a  study  are  Interpreted  In  a 
broadening  series  of  contexts.  These  contexts  of  Interpretation  are  the  statistical  frame¬ 
work  In  which  the  result  Is  derived,  the  purpose  of  the  particular  study,  and  the 
knowledge  In  the  field  of  research  [15],  The  representativeness  of  the  sampling  analyzed 
In  a  study  qualifies  the  extrapolation  of  the  results  to  other  environments  [20).  Several 
follow-up  activities  contribute  to  the  Impact  of  a  study:  presentlng/publlshlng  the 
results  for  feedback,  replicating  the  experiment  [33,  40],  and  actually  applying  the 
results  by  modifying  methods  for  software  development,  maintenance,  management,  and 
research. 

4.  Classification  of  Analyses 

Several  Investigators  have  published  studies  In  the  four  general  scopes  of  examina¬ 
tion:  blocked  subject-project,  replicated  project,  multi-project  variation,  or  single  pro¬ 
ject.  The  following  sections  cite  studies  from  each  of  these  categories.  Note  that  sur- 


veys  on  experimental  methodology  In  empirical  studies  Include  [35,  96,  74].  Each  of  the 
sections  first  discusses  one  experiment  In  moderate  depth,  using  Italicized  keywords  from 
the  framework  for  experimentation,  and  then  chronologically  presents  an  overview  of 
several  others  In  the  category. 

4.1.  Blocked  Subject-Project  Studies 

With  a  motivation  to  Improve  and  better  understand  unit  testing,  [24]  conducted  a 
study  whose  purpose  was  to  characterize  and  evaluate  the  processes  (l.e.,  objects)  of  code 
reading,  functional  testing,  and  structural  testing  from  the  perspective  of  the  developer. 
The  testing  processes  were  examined  In  a  blocked  subject-project  scope,  where  74  stu¬ 
dent  through  professional  programmers  (from  the  programmer  domain)  tested  four  unit- 
size  programs  (from  the  program  domain)  In  a  replicated  fractional  factorial  design.  Ob¬ 
jective  measurement  of  the  testing  processes  was  In  several  criteria  areas:  fault  detection 
effectiveness,  fault  detection  cost,  and  classes  of  faults  detected.  Experiment  prepara¬ 
tion  included  a  pilot  study  [63],  execution  Incorporated  both  manual  and  automated 
monitoring  of  testing  activity,  and  analysis  used  analysis  of  variance  methods  [33,  90]. 
The  major  results  (In  the  interpretation  context  of  the  study  purpose)  Included  1)  with 
the  professionals,  code  reading  detected  more  software  faults  and  had  a  higher  fault 
detection  rate  than  did  the  other  methods;  2)  with  the  professionals,  functional  testing 
detected  more  faults  than  did  structural  testing,  but  they  were  not  different  in  fault 
detection  rate;  3)  with  the  students,  the  three  techniques  were  not  different  In  perfor¬ 
mance,  except  that  structural  testing  detected  fewer  faults  than  did  the  others  In  one 
study  phase;  and  4)  overall,  code  reading  detected  more  Interface  faults  and  functional 


testing  detected  more  control  faults  than  did  the  other  methods.  A  major  result  (In  the 
interpretation  context  of  the  field  of  research)  Is  that  the  study  suggests  that  non- 
execution  based  fault  detection,  as  In  code  reading,  Is  at  least  as  effective  as  on-line 
methods.  The  particular  programmers  and  programs  sampled  qualify  the  extrapolation 
of  the  results.  The  impact  of  the  study  Is  an  advancement  In  the  understanding  of 
effective  software  testing  methods. 

In  order  to  understand  program  debugging,  [57]  evaluated  several  related  factors. 
Including  effect  of  debugging  aids,  effect  of  fault  type,  and  effect  of  particular  program 
debugged  from  the  perspective  of  the  developer  and  maintained  Thirty  experienced 
programmers  Independently  debugged  one  of  four  one-page  programs  that  contained  a 
single  fault  from  one  of  three  classes.  The  major  results  of  these  studies  were  1)  debug¬ 
ging  Is  much  faster  If  the  programmer  has  had  previous  experience  with  the  program,  2) 
assignment  bugs  were  harder  to  find  than  other  kinds,  and  3)  debugging  aids  did  not 
seem  to  help  programmers  debug  faster.  Consistent  results  were  obtained  when  the 
study  was  conducted  on  ten  additional  experienced  programmers  [58],  These  results  and 
the  Identification  of  possible  "principles”  of  debugging  contribute  to  the  understanding 
of  debugging  methodology. 

In  order  to  improve  experimental  methodology  and  Its  application,  [110]  evaluated 
programmers’  ability  to  understand  and  modify  a  program  from  the  perspective  of  the 
developer  and  modifier.  Various  measures  of  programmer  understanding  were  calculat¬ 
ed,  In  a  series  of  factorial  design  experiments,  on  groups  of  16  -  48  university  students 
performing  tasks  on  two  small  programs.  The  study  emphasized  the  need  for  well- 
structured  and  weil-documented  pre-grams,  and  provided  valuable  testimony  on  and 


worked  toward  a  suitable  experimentation  methodology. 

In  order  to  assess  the  Impact  of  language  features  on  the  programming  process,  [53] 
characterized  the  relationship  of  language  features  to  software  reliability  from  the  per¬ 
spective  of  the  developer.  Based  on  an  analysis  of  the  deficiencies  In  a  programming 
language,  nine  different  features  were  modified  to  produce  a  new  version.  Fifty-one  ad¬ 
vanced  students  were  divided  Into  two  groups  and  asked  to  complete  implementations  of 
two  small  but  sophisticated  programs  (75-200  tine)  In  the  original  language  and  Its 
modified  version.  The  redesigned  features  In  the  two  languages  were  contrasted  In  pro¬ 
gram  fault  frequency,  type,  and  persistence.  The  experiment  Identified  several 
language-design  decisions  that  significantly  affected  reliability,  which  contributes  to  the 
understanding  of  language  design  for  reliable  software. 

In  order  to  understand  the  unit  testing  process  better,  [60]  evaluated  a  reading 
technique  and  functional  and  “selective"  testing  (a  composite  approach)  from  the  per¬ 
spective  of  the  developer.  Thirty-nine  university  students  applied  the  techniques  to 
three  unit-size  programs  in  a  Latin  square  design.  Functional  and  “selective”  testing 
were  equally  effective  and  both  superior  to  the  reading  technique,  which  contributed  to 
our  understanding  of  testing  methodology. 

In  order  to  Improve  and  better  understand  the  maintenance  process,  [43]  conducted 
two  experiments  to  evaluate  factors  that  Influence  two  aspects  of  software  maintenance, 
program  understanding  and  modification,  from  the  perspective  of  the  developer  and 
malntalner.  Thirty-six  Junior  through  advanced  professional  programmers  In  each  ex¬ 
periment  examined  three  classes  of  small  (36  -  57  source  line)  programs  in  a  factorial 
design.  The  factors  examined  Include  control  flow  complexity,  variable  name  mnemonl- 
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city,  type  of  modification,  degree  of  commenting,  and  the  relationship  of  programmer 
performance  to  various  complexity  metrics.  In  [44]  they  continued  the  Investigation  of 
how  software  characteristics  relate  to  psychological  complexity,  and  presented  a  third 
experiment  to  evaluate  the  ability  of  54  professional  programmers  to  detect  program 
bugs  in  three  programs  In  a  factorial  design.  The  series  of  experiments  showed  that 
software  science  [59]  and  cyclomatlc  complexity  [7l]  measures  are  related  to  the 
difficulty  experienced  by  programmers  In  locating  errors  In  code. 

In  order  to  improve  and  better  understand  program  debugging,  [108]  evaluated  the 
theory  that  "programmers  use  ’slicing’  (stripping  away  a  program’s  statements  that  do 
not  Influence  a  given  variable  at  a  given  statement)  when  debugging"  from  the  perspec¬ 
tive  of  the  developer,  malntalner,  and  researcher.  Twenty-one  university  graduate  stu¬ 
dents  and  programming  staff  debugged  a  fault  In  three  unit-size  (75  -  150  source  line) 
programs  In  a  non-parametrlc  design.  The  study  results  supported  the  slicing  theory, 
that  Is,  programmers  during  debugging  routinely  partitioned  programs  Into  a  coherent, 
discontiguous  piece  (or  slice).  The  results  advance  the  understanding  of  software  debug¬ 
ging  methodology. 

In  order  to  Improve  design  techniques,  [87]  evaluated  flowcharts  and  program 
design  languages  (PDL)  from  the  perspective  of  the  developer.  Twenty-two  graduate 
students  designed  two  small  (approximately  1000  source  line)  projects,  one  using 
flowcharts  and  the  other  using  PDL.  Overall,  the  results  suggested  that  design  perfor¬ 
mance  and  designer-programmer  communication  were  better  for  projects  using  PDL. 
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In  order  to  validate  a  theory  of  programming  knowledge,  [101]  conducted  two  stu¬ 
dies,  using  139  novices  and  41  professional  programmers,  to  evaluate  programmer 
behavior  from  the  perspective  of  the  researcher.  The  theory  was  that  programming 
knowledge  contained  programming  plans  (generic  program  fragments  representing  com¬ 
mon  actions  sequences)  and  rules  of  programming  discourse  (conventions  used  In  com¬ 
posing  plans  Into  programs).  The  results  support  the  existence  and  use  of  such  plans 
and  rules  by  both  novice  and  advanced  programmers. 

Other  blocked  subject-project  studies  Include  [82,  1 12]. 

4.2.  Replicated  Project  Studies 

With  a  motivation  to  assess  and  better  understand  team  software  development 
methodologies,  [15[  conducted  a  study  whose  purpose  was  to  characterize  and  evaluate 
the  development  processes  (l.e.,  objects)  of  a  a)  dlsclpllned-methodology  team  approach, 
b)  ad  hoc  team  approach,  and  c)  ad  hoc  Individual  approach  from  the  perspective  of  the 
developer  and  project  manager.  The  development  processes  were  examined  in  a  repli¬ 
cated  project  scope.  In  which  advanced  university  students  comprising  seven  three- 
person  teams,  six  three-person  teams,  and  six  Individuals  (from  the  programmer  domain) 
used  the  approaches,  respectively.  They  separately  developed  a  small  (000  -  2200  line) 
compiler  (from  the  program  domain)  In  a  non-parametrlc  design.  Objective  measure¬ 
ment  of  the  development  approaches  was  In  several  criteria  areas:  number  of  changes, 
number  of  program  runs,  program  data  usage,  program  data  coupllng/blndlng,  static 
program  slze/complexlty  metrics,  language  usage,  and  modularity.  Experiment  prepara¬ 
tion  Included  presentation  of  relevant  material  [68,  7,  34],  execution  Included  automated 


monitoring  of  on-line  development  activity  and  analysis  used  non-parametric  comparison 
methods.  The  major  results  (In  the  interpretation  context  of  the  study  purpose)  Includ¬ 
ed  1)  the  methodological  discipline  was  a  key  Influence  on  the  general  efficiency  of  the 
software  development  process;  2)  the  disciplined  team  methodology  significantly  reduced 
the  costs  of  software  development  as  reflected  In  program  runs  and  changes;  and  3)  the 
examination  of  the  effect  of  the  development  approaches  was  accomplished  by  the  use  of 
quantitative,  objective,  unobtrusive,  and  automatable  process  and  product  metrics.  A 
major  result  (in  the  interpretation  context  of  the  field  of  research)  Is  that  the  study  sup¬ 
ports  the  belief  that  Incorporating  discipline  In  software  development  reflects  positively 
on  both  the  development  process  and  Anal  product.  The  particular  programmers  and 
program  sampled  qualify  the  extrapolation  of  the  results.  The  impact  of  the  study  Is  an 
advancement  In  the  understanding  of  software  development  methodologies  and  their 
evaluation. 

In  order  to  Improve  the  design  and  Implementation  processes,  [84]  evaluated  system 
modularity  from  the  perspective  of  the  developer.  Twenty  university  undergraduates 
each  developed  one  of  four  different  types  of  Implementations  for  one  of  five  different 
small  modules.  Then  each  of  the  modules  were  combined  with  others  to  form  several 
versions  of  the  whole  system.  The  major  results  suggested  that  minor  effort  was  re¬ 
quired  In  assembling  the  systems  and  that  major  system  changes  can  be  confined  to 
small,  well-defined  subsystems.  The  results  support  the  Ideas  on  formal  specifications 
and  modularity  discussed  In  [83,  85]  and  advance  the  understanding  of  design  methodol- 


In  order  to  assess  the  Impact  of  static  typing  of  programming  languages  In  the  de¬ 
velopment  process,  [54]  evaluated  the  use  of  a  statically  typed  language  (having  Integers 
and  strings)  and  a  "typeless"  language  (e.g.,  arbitrary  subscripting  of  memory)  from  the 
perspective  of  the  developer.  Thirty-eight  students  programmed  a  small  (48  -  297 
source  line)  problem  In  both  languages,  with  half  doing  It  In  each  order.  The  two 
languages  were  compared  In  the  resulting  program  faults,  the  number  of  runs  containing 
faults,  and  the  relation  of  subject  experience  to  fault  proneness.  The  major  result  was 
that  the  use  of  a  statically  typed  language  can  Increase  programming  reliability,  which 
assists  In  the  design  and  use  of  programming  languages. 

In  order  to  Improve  program  composition,  comprehension,  debugging,  and 
modification,  [98]  evaluated  the  use  of  detailed  flowcharts  In  these  tasks  from  the  per¬ 
spective  of  the  developer,  malntalner,  modifier,  and  researcher.  Groups  of  53  -  70  no¬ 
vice  through  Intermediate  subjects.  In  a  series  of  five  experiments,  performed  various 
tasks  using  small  programs.  No  significant  differences  were  found  between  groups  that 
used  and  those  that  did  not  use  flowcharts,  questioning  the  merit  of  using  detailed 
flowcharts. 

In  order  to  Improve  and  better  understand  the  unit  testing  process,  [79]  evaluated 
the  techniques  of  three-person  walk-throughs,  functional  testing,  and  a  control  group 
from  the  perspective  of  the  developer.  Fifty-nine  Junior  through  advanced  professional 
programmers  applied  the  techniques  to  test  a  small  (100  source  line)  but  nontrivial  pro¬ 
gram.  The  techniques  were  not  different  In  the  number  of  faults  they  detected,  all  pair¬ 
ings  of  techniques  were  superior  to  single  techniques,  and  code  reviews  were  less  cost- 
effect'.ve  than  the  others.  These  results  assist  In  the  selection  of  appropriate  software 


testing  techniques. 


In  order  to  validate  a  particular  metric  family,  [17]  evaluated  the  ability  of  a  pro¬ 
posed  metric  family  to  explain  differences  In  system  development  methodologies  and  sys¬ 
tem  changes  from  the  perspective  of  the  developer,  project  manager,  and  researcher. 
The  metrics  were  applied  to  19  versions  of  a  small  (600  -  2200)  compiler,  which  were 
developed  by  teams  of  advanced  university  students  using  three  different  development 
approaches  (see  the  first  study  [15]  described  In  this  section).  The  major  results  Includ¬ 
ed  1)  the  metrics  were  able  to  differentiate  among  projects  developed  with  different  de¬ 
velopment  methodologies;  and  2)  the  differences  among  Individuals  had  a  large  effect  on 
the  relationships  between  the  metrics  and  aspects  of  system  development.  These  results 
suggest  Insights  Into  the  formulation  and  appropriate  use  of  software  metrics. 

In  order  to  Improve  the  understanding  of  why  software  errors  occur,  [65]  character¬ 
ized  programmer  misconceptions,  cognitive  strategies,  and  their  manifestations  as  bugs 
In  programs  from  the  perspective  of  the  developer  and  researcher.  Two-hundred-four 
novice  programmers  separately  attempted  Implementations  of  an  elementary  program. 
The  results  supported  the  programmers’  Intended  use  of  ’’programming  plans”  [100]  and 
revealed  that  most  people  preferred  a  read-process  strategy  over  a  process-read  strategy. 
The  results  advance  the  understanding  of  how  Individuals  write  programs,  why  they 
sometimes  make  errors,  and  what  programming  language  constructs  should  be  available. 

In  order  to  understand  the  effect  of  coding  conventions  on  program  comprehensibili¬ 
ty,  [73]  conducted  a  study  to  evaluate  the  relationship  between  Indentation  levels  and 
program  comprehension  from  the  perspective  of  the  developer.  Elghty-slx  novice 
through  professional  subjects  answered  questions  about  one  of  seven  program  variations 


with  different  level  and  type  of  Indentation.  The  major  result  was  that  an  Indentation 
level  of  two  or  four  spaces  was  preferred  over  zero  or  six. 

In  order  to  improve  software  development  approaches,  [29]  characterized  and 
evaluated  the  prototyping  and  specifying  development  approaches  from  the  perspective 
of  the  developer,  project  manager,  and  user.  Seven  two-  and  three-person  teams,  con-  i 

t 

i 

slstlng  of  university  graduate  students,  developed  versions  of  the  same  application  soft¬ 
ware  system  (2000  -  4000  line);  four  teams  used  a  requirement/design  specifying  ap- 

t 

t 

proach  and  three  teams  used  a  prototyping  approach.  The  systems  developed  by  proto-  ] 

typing  were  smaller,  required  less  development  effort,  and  were  easier  to  use.  The  sys¬ 
tems  developed  by  specifying  had  more  coherent  designs,  more  complete  functionality,  ] 

i 

and  software  that  was  easier  to  Integrate.  These  results  contribute  to  the  understanding 

of  the  merits  and  appropriateness  of  software  development  approaches.  I 

i 

In  order  to  validate  the  theoretical  model  for  N-verslon  programming  [86],  [67,  3] 

I 

conducted  a  study  to  evaluate  the  effectiveness  of  N-verslon  programming  for  reliability 
from  the  perspective  of  the  customer  and  user.  N-verslon  programming  uses  a  high-level  ' 

driver  to  connect  several  separately  designed  versions  of  the  same  system,  the  systems 
"vote”  on  the  correct  solution,  and  the  solution  provided  by  the  majority  of  the  systems 
Is  output.  Twenty-seven  graduate  students  were  asked  to  Independently  design  an  800 
source  line  system.  The  factors  examined  included  Individual  system  reliability,  total 
N-verslon  system  reliability,  and  classes  of  faults  that  occurred  In  systems  simultaneous¬ 
ly.  The  major  result  was  that  the  assumption  of  Independence  of  the  faults  In  programs 
Is  not  Justified,  and  therefore,  the  reliability  of  the  combined  "voting"  system  may  not 
be  as  high  as  given  by  the  model. 
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In  order  to  Improve  and  better  understand  software  development  approaches,  [94] 
characterized  and  evaluated  the  Cleanroom  development  approach  [47,  46],  In  which 
software  Is  developed  without  execution  (l.e.,  completely  off-line),  from  the  perspective 
of  the  developer,  project  manager,  and  customer.  Fifteen  three-person  teams  of  ad¬ 
vanced  university  students  separately  developed  a  small  system  (800  -  2300  source  line); 
ten  teams  used  Cleanroom  and  five  teams  used  a  traditional  development  approach  in  a 
non-parametrlc  design.  The  major  results  Included  1)  most  developers  using  the  Clean¬ 
room  approach  were  able  to  build  systems  without  program  execution;  and  2)  the  Clean¬ 
room  teams’  products  met  system  requirements  more  completely  and  succeeded  on  more 
operational  test  cases  than  did  those  developed  with  a  traditional  approach.  The  results 
suggest  the  feasibility  of  complete  off-line  development,  as  In  Cleanroom,  and  advance 
the  understanding  of  software  development  methodology. 

Other  replicated  project  studies  Include  [37,  5,  63]. 

4.3.  Multi-Project  Variation  Studies 

With  a  motivation  to  Improve  the  understanding  of  resource  usage  during  software 
development,  [4]  conducted  a  study  whose  purpose  was  to  predict  development  cost  by 
using  a  particular  model  (l.e.,  object )  and  to  evaluate  It  from  the  perspective  of  the  pro¬ 
ject  manager,  corporate  manager,  and  researcher.  The  particular  model  generation 
method  was  examined  In  a  multi-project  scope,  with  baseline  data  from  18  large  (2500  - 
100,000  source  line)  software  projects  In  the  NASA  S.E.L.  production  environment  (rrom 
the  program  domain),  In  which  teams  contained  from  two  to  ten  programmers  (from  the 
programmer  domain)  [10,  11,  38,  91].  The  study  design  Incorporated  multivariate 


16 


methods  to  parameterize  the  model.  Objective  and  subjective  measurement  of  the  pro¬ 
jects  was  based  on  21  criteria 2  In  three  areas:  methodology,  complexity,  and  personnel 
experience.  Study  preparation  Included  preliminary  work  [52j,  execution  Included  an  es¬ 
tablished  set  of  data  collection  forms  [10],  and  analysis  used  forward  multivariate  regres¬ 
sion  methods.  The  major  results  (In  the  interpretation  context  of  the  study  purpose)  in¬ 
cluded  1)  the  estimation  of  software  development  resource  usage  improved  by  consider¬ 
ing  a  set  of  both  base-line  and  customization  factors;  2)  the  application  In  the  NASA 
environment  of  the  proposed  model  generation  method,  which  considers  both  types  of 
factors,  produced  a  resource  usage  estimate  for  a  future  project  within  one  standard  de¬ 
viation  of  the  actual;  and  3)  the  confirmation  of  the  NASA  S.E.L.  formula  that  the  cost 
per  line  of  reusing  code  is  20%  of  that  of  developing  new  code.  A  major  result  (in  the 
interpretation  context  of  the  field  of  research)  Is  that  the  study  highlights  the  difference 
of  each  software  development  environment,  which  Influences  the  use  of  resource  estima¬ 
tion  models.  The  particular  programming  environment  and  projects  sampled  qualify  the 
extrapolation  of  the  results.  The  impact  of  the  study  is  an  advancement  In  the  under¬ 
standing  of  estimating  software  development  resource  expenditure. 

In  order  to  assess,  manage,  and  Improve  multi-project  environments,  [28,  26,  106, 
13,  38,  18,  82,  109,  97,  105]  characterized,  evaluated,  and/or  predicted  the  effect  of 
several  factors  from  the  perspective  of  the  developer,  modifier,  project  manager,  and 
corporate  manager.  All  the  studies  examined  moderate  to  large  projects  from  produc- 

2  Twenty-one  factors  were  selected  after  examining  a  total  of  82  factors  that  possi¬ 
bly  contributed  to  project  resource  expenditure.  Including  36  from  [108]  and  16  from 


tlon  environments.  The  relationships  Investigated  were  among  various  factors.  Including 
structured  programming,  personnel  background,  development  process  and  product  con¬ 
straints,  project  complexity,  human  and  computer  resource  consumption,  error-prone 
software  Identification,  error/change  distributions,  data  coupllng/blndlng,  project  dura¬ 
tion,  staff  size,  degree  of  management  control,  and  productivity.  These  studies  have 
provided  Increased  project  visibility,  greater  understanding  of  classes  of  factors  sensitive 
to  project  performance,  awareness  of  the  need  for  project  measurement,  and  efforts  for 
standardization  of  definitions.  Analysis  has  begun  on  Incorporating  project  variation  In¬ 
formation  Into  a  management  tool  [18,  23j. 

In  order  to  Improve  and  better  understand  the  software  maintenance  process,  [104] 
conducted  an  experiment  to  evaluate  the  relationship  between  the  rate  of  maintenance 
repair  and  various  product  and  process  metrics  from  the  perspective  of  the  developer, 
user,  and  the  project  manager.  A  total  of  447  small  (up  to  600  statements)  commercial 
and  clerical  Cobol  programs  from  one  Australian  organization  and  two  U.S.  organiza¬ 
tions  were  analyzed.  The  product  and  process  metrics  Included  program  complexity, 
programming  style,  programmer  quality,  and  number  of  system  releases.  The  major 
results  were  1)  In  the  Australian  organization,  program  complexity  and  programming 
style  significantly  affected  the  maintenance  repair  rate;  and  2)  In  the  U.S.  organizations, 
the  number  of  times  a  system  was  released  significantly  affected  the  maintenance  repair 
rate. 

In  order  to  Improve  the  software  maintenance  process,  [l]  evaluated  operational 
faults  from  the  perspective  of  the  user,  customer,  project  manager,  and  corporate 
manager.  The  fault  history  for  nine  large  production  products  (e.g..  operating  system 


releases  or  their  major  components)  was  empirically  modeled.  He  developed  an  ap¬ 


proach  for  estimating  whether  and  under  what  circumstances  preventively  fixing  faults 
In  operational  software  In  the  field  was  appropriate.  Preventively  fixing  faults  consists 
of  Installing  fixes  to  faults  that  have  yet  to  be  discovered  by  particular  users,  but  have 
been  discovered  by  the  vendor  or  other  users.  The  major  result  Is  that  for  the  typical 
user,  corrective  service  Is  a  reasonable  way  of  dealing  with  most  faults  after  the  code  has 
been  In  use  for  a  fairly  long  period  of  time,  while  preventively  fixing  hlgh-rate  faults  Is 
advantageous  during  the  time  Immediately  following  release. 

In  order  to  assess  the  effectiveness  of  the  testing  process,  [31]  evaluated  estimations 
of  the  number  of  residual  faults  In  a  system  from  the  perspective  of  the  customer, 
developer,  and  project  manager.  The  study  was  based  on  fault  data  collected  from 
three  large  (2000  -  6000  module)  systems  developed  in  the  Hughes-Fullerton  environ¬ 
ment.  The  study  partitioned  the  faults  based  on  severity  and  analyzed  the  differences  In 
estimates  of  remaining  faults  according  to  stage  of  testing.  Insights  were  gained  Into  re¬ 
lationships  between  fault  detection  rates  and  residual  faults. 

4.4.  Single  Project  Studies 

With  a  motivation  to  Improve  software  development  methodology,  [8]  conducted  a 
study  whose  purpose  was  to  characterize  the  process  (I.e.,  object)  of  Iterative  enhance¬ 
ment  In  conjunction  with  a  top-down,  stepwise  refinement  development  approach  from 
the  perspective  of  the  developer.  The  development  process  was  examined  In  a  single 
project  scope,  where  the  authors,  two  experienced  Individuals  (from  the  programmer 
domain),  built  a  17,000  line  compiler  (from  the  program  domain).  The  study  design  In- 


corporated  descriptive  methods  to  capture  system  evolution.  Objective  measurement  of 
the  system  was  In  several  criteria  areas:  size,  modularity,  local/global  data  usage,  and 
data  blndlng/coupllng  [62,  102).  Study  preparation  Included  language  design  [9],  execu¬ 
tion  Incorporated  static  analysis  of  system  snapshots,  and  analysis  used  descriptive 
statistics.  The  results  (In  the  interpretation  context  of  the  statistical  framework)  Includ¬ 
ed  1)  the  percentage  of  global  variables  decreased  over  time  while  the  percentage  of  ac¬ 
tual  vs.  possible  data  couplings  across  modules  Increased,  suggesting  the  usage  of  global 
data  became  more  appropriate  over  time;  and  2)  the  number  of  procedures  and  func¬ 
tions  rose  over  time  while  the  number  of  statements  per  procedure  or  function  de¬ 
creased,  suggesting  Increased  modularity.  The  major  result  of  the  study  (In  the  in¬ 
terpretation  context  of  the  study  purpose)  was  that  the  Iterative  enhancement  technique 
encouraged  the  development  of  a  software  product  that  had  several  generally  desirable 
aspects  of  system  structure.  A  major  result  (In  the  interpretation  context  of  the  held  of 
research)  Is  that  the  study  demonstrates  the  feasibility  of  Iterative  enhancement.  The 
particular  programming  team  and  project  examined  qualify  the  extrapolation  of  the 
results.  The  impact  of  the  study  Is  an  advancement  In  the  understanding  of  software 
development  approaches. 

In  order  to  Improve,  better  understand,  and  manage  the  software  development  pro¬ 
cess,  [6]  evaluated  the  effect  of  applying  chief  programming  teams  and  structured  pro¬ 
gramming  In  system  development  from  the  perspective  of  the  user,  developer,  project 
manager,  and  corporate  manager.  The  large  (83,000  line)  system,  known  as  "The  New 
York  Times  Project."  and  was  developed  by  a  team  of  professionals  organized  as  a  chief 
programmer  team,  using  structured  code,  top  down  design,  walk-throughs,  and  program 
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libraries.  Several  benefits  were  Identified,  Including  reduced  development  time  and  cost, 
reduced  time  In  system  Integration,  and  reduced  fault  detection  In  acceptance  testing 
and  field  use.  The  results  of  the  study  demonstrated  the  feasibility  of  the  chief  pro¬ 
grammer  team  concept  and  the  accompanying  methodologies  In  a  production  environ¬ 
ment. 

In  order  to  Improve  their  development  environments  through  Increased  understand¬ 
ing,  [49,  14,  2,  81,  19]  each  conducted  single  project  studies  to  characterize  the  errors 
and  changes  made  during  a  development  project.  They  examined  the  development  of  a 
moderate  to  large  software  project,  done  by  a  multi-person  team.  In  a  production  en¬ 
vironment.  They  analyzed  the  frequency  and  distribution  of  errors  during  development 
and  their  relationship  with  several  factors.  Including  module  size,  software  complexity, 
developer  experience,  method  of  detection  and  Isolation,  effort  for  Isolation  and  correc¬ 
tion,  phase  of  entrance  Into  the  system  and  observance,  reuse  of  existing  design  and 
code,  and  role  of  the  requirements  document.  Such  analyses  have  produced  fault 
categorization  schemes  and  have  been  useful  in  understanding  and  Improving  a  develop¬ 
ment  environment. 

In  order  to  Improve  design  methodology,  [55,  27]  examined  a  ground-support  sys¬ 
tem  written  In  Ada3  to  characterize  the  use  of  Ada  packages  from  the  perspective  of  the 
developer.  Four  professional  programmers  developed  a  project  of  10,000  source  lines  of 
code.  Factors  such  as  how  package  use  affected  the  ease  of  system  modification  and 

3  Ada  Is  a  trademark  of  the  Department  of  Defense. 


how  to  measure  module  change  resistance  were  Identified,  as  well  as  how  these  observa- 
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tlons  related  to  aspects  of  the  development  and  training.  The  major  results  were  l) 
several  measures  of  Ada  programs  were  developed,  and  2)  there  was  a  Indication  that  a 
lot  of  training  will  be  necessary  If  we  are  to  expect  the  facilities  of  Ada  to  be  properly 
used. 

In  order  to  assess  and  Improve  software  testing  methodology,  [21,  88]  characterized 
and  evaluated  the  relationship  between  system  acceptance  tests  and  operational  usage 
from  the  perspective  of  the  developer,  project  manager,  customer,  and  researcher.  The 
execution  coverage  of  functionally  generated  acceptance  test  cases  and  a  sample  of 
operational  usage  cases  was  monitored  for  a  medium-size  (10,000  line)  software  system 
developed  In  a  production  environment.  The  results  calculated  that  Q4%  of  the  pro¬ 
gram  statements  were  executed  during  system  operation  and  that  the  acceptance  test 
cases  corresponded  reasonably  well  to  the  operational  usage.  The  results  give  Insights 
Into  the  relationships  among  structural  coverage,  fault  detection,  system  testing,  and 
system  usage. 


5.  Problem  Areas  in  Experimentation 

The  following  sections  Identify  several  problem  areas  of  experimentation  In  software 
engineering.  These  areas  may  serve  as  guidelines  In  the  performance  of  future  studies. 
After  mentioning  some  overall  observations,  cautions  In  each  of  the  areas  of  experiment 
definition,  planning,  operation,  and  Interpretation  are  discussed. 
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5.1.  Experimentation  Overall 

There  appears  to  be  no  “universal  model"  or  “silver  bullet"  in  software  engineering. 
There  are  an  enormous  number  of  factors  that  differ  across  environments.  In  terms  of 
desired  cost/quality  goals,  methodology,  experience,  problem  domain,  constraints,  etc. 
(100,  26,  4,  13,  28).  This  results  In  every  software  development/maintenance/...  environ¬ 
ment  being  different.  Another  area  of  wide  variation  Is  the  many-to-one  differential  In 
human  performance  [17,  45,  24).  The  particular  Individuals  examined  in  an  empirical 
study  can  make  an  enormous  difference.  Among  other  considerations,  these  variations 
suggest  that  metrics  need  to  be  validated  for  a  particular  environment  and  a  particular 
person  to  show  that  they  capture  what  Is  Intended  [17,  18].  Thus,  experimental  studies 
should  consider  the  potentially  vast  differences  among  environments  and  people. 


5.2.  Experiment  Definition 

In  the  definition  of  the  purpose  for  the  experiment,  the  formulation  of  Intuitive 
problems  Into  precisely  stated  goals  Is  a  nontrivial  task  [20,  22].  Defining  the  purpose  of 


a  study  often  requires  the  articulation  of  what  Is  meant  by  "software  quality."  The 


many  Interpretations  and  perceptions  of  quality  [32,  39,  72]  highlight  the  need  for  con¬ 
sidering  whose  perspective  of  quality  Is  being  examined.  Thus,  a  precise  specification  of 
the  problem  to  be  Investigated  Is  a  major  step  toward  Its  solution. 


5.3.  Experiment  Planning 

Experimental  planning  should  have  a  horizon  beyond  a  first  experiment.  Con¬ 
trolled  studies  may  be  used  to  focus  on  the  effect  of  certain  factors,  while  their  results 


may  be  confirmed  In  replications  [92,  98,  101,  110,  57,  58,  44,  43,  24]  and/or  larger  case 
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studies  [4,  15].  When  designing  studies,  consider  that  a  combination  of  factors  may  be 
effective  as  a  “critical  mass,”  even  though  the  particular  factors  may  be  Ineffective  when 
treated  In  Isolation  [15,  I05j.  Note  that  formal  designs  and  the  resulting  statistical 
robustness  are  desirable,  but  we  should  not  be  driven  exclusively  by  the  achievement  of 
statistical  significance.  Common  sense  must  be  maintained,  which  allows  us,  for  exam¬ 
ple,  to  experiment  Just  to  help  develop  hypotheses  [19,  109].  Thus,  the  experimental 
planning  process  should  Include  a  series  of  experiments  for  exploration,  verification,  and 
application. 

5.4.  Experiment  Operation 

The  collection  of  the  required  data  constitutes  the  primary  result  of  the  study 
operation  phase.  The  data  must  be  carefully  defined,  validated,  and  communicated  to 
ensure  Its  consistent  Interpretation  by  all  persons  associated  with  the  experiment:  sub¬ 
jects  under  observation,  experimenters,  and  literature  audience  [IS],  There  have  been 
papers  In  the  literature  that  do  not  define  their  data  well  enough  to  enable  a  comparison 
of  results  across  many  projects  and  environments.  We  have  often  contacted  the  experi¬ 
menter  to  discover  that  we  are  measuring  different  things.  Thus,  the  experimenter 
should  be  cautious  about  the  definition,  validation,  and  communication  of  data,  since 
they  play  a  fundamental  role  In  the  experimental  process. 

5.5.  Experiment  Interpretation 

The  appropriate  presentation  of  results  from  experiments  contributes  to  their 
correct  Interpretation.  Experimental  results  need  to  be  qualified  by  the  particular  sam¬ 
ples  (e.g.,  programmers,  programs)  analyzed  [20].  The  extrapolation  o'  results  from  a 


particular  sample  must  consider  the  representativeness  of  the  sample  to  other  environ¬ 
ments  [41,  111,  106,  86,  4,  28] .  The  visibility  of  the  experimental  results  In  professional 
forums  and  the  open  literature  provides  valuable  feedback  and  constructive  criticism. 
Thus,  the  presentation  of  experimental  results  should  Include  appropriate  qualification 
and  adequate  exposure  to  support  their  proper  Interpretation. 

6.  Conclusion 

Experimentation  In  software  engineering  supports  the  advancement  of  the  field 
through  an  Iterative  learning  process.  The  experimental  process  has  begun  to  be  applied 
In  a  multiplicity  of  environments  to  study  a  variety  of  software  technology  areas.  From 
the  studies  presented.  It  Is  clear  that  experimentation  has  proven  effective  In  providing 
Insights  and  furthering  our  domain  of  knowledge  about  the  software  process  and  pro¬ 
duct.  In  Tact,  there  Is  a  learning  process  In  the  experimentation  approach  Itself,  as  has 
been  shown  In  this  paper. 

We  have  described  a  framework  for  experimentation  to  provide  a  structure  for 
presenting  previous  studies.  We  also  recommend  the  framework  as  a  mechanism  to  fa¬ 
cilitate  the  definition,  planning,  operation,  and  Interpretation  of  past  and  future  studies. 
The  problem  areas  discussed  are  meant  to  provide  some  useful  recommendations  for  the 
application  of  the  experimental  process  In  software  engineering.  The  experimental 
framework  cannot  be  used  In  a  vacuum;  the  framework  and  the  lessons  learned  comple¬ 
ment  one  another  and  should  be  used  In  a  synergistic  fashion.  This  work  contributes  to 
the  understanding  and  advancement  of  experimentation  In  software  engineering. 
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Figure  1.  Summary  of  the 


framework  of  experimentation. 
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Figure  2.  Studv  definition  example. 
Definition  element  I 


example 


Motivation 


Object 


Perspective 


Domain:  programmer 


Domain:  program 


Scope 


To  improve  the  unit  testing  process. 


characterize  and  evaluate 


the  processes  of  functional  and  structural  testing 


from  the  perspective  of  the  developer 


as  they  are  applied  by  experienced  programmers 


to  unit-size  software 


in  a  blocked  subject-project  study. 


Figure  3.  Experimental  scopes. 
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